p8/60864 

' 73492DMW 



A ^IGITAIi IMAGING DEVICE OPTIMIZED FOR COLOR 1 

jPERFORMANCsJ 

FIELD OF THE INVENTION 

This invention pertains to the field of 
digital imaging and, more particularly, to the 
optimization of the color performance of digital 
imaging devices, such as digital cameras and scanners. 

BACKGROUND OF THE INVENTION 



Correction matrices are useful in a variety 
of color imaging applications to effect color 
conversion or correction. For instance, a conversion 
matrix is used to convert red, green, and blue video 
signals into Y (luminance) and I, Q (chrominance) 
signals. A color correction matrix is used to correct 
the spectral sensitivities of a video camera for the 
chromaticities of the phosphor set of the particular 
display in use. Another use is with film- to-video 
conversion, a process in which a color correction 
matrix operates on the film scanning signals to correct 
the film colorimetry for video display. While these 
systems were typically analog systems, matrix 
processing is particularly adapted to a digital 
environment . 

Continuing advances in semiconductor 
technology in areas such as digital memory, digital 
application-specific integrated circuits (ASICs) and 
charge-coupled device (CCD) imagers have made possible 
the introduction in recent years of digital electronic 
cameras. Evolution of this product segment will be 
driven by ever increasing consumer demands for better 
performance in such areas as resolution, photographic 
speed, and color reproduction. In the area of color 
reproduction it is desirable to select an optimum set 
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of spectral characteristics for the CCD imager. The 
prior art (for example, as described in Color Science 
in Television and Display Systems by W.N. Sproson, 
published by Adam Hilger Ltd, 1983), teaches that one 
5 step toward the goal of good color reproduction is to 
choose a set of spectral characteristics for the camera 
which are as close as possible to the spectral 
characteristics of the intended display device. In the 
aforementioned Sproson text, a color cathode-ray tube 

10 (CRT) is used as an example of a typical display device 
where the defining spectral characteristics are easily 
derived by someone skilled in the art from a knowledge 
of the CRT's phosphor chromaticities and white-point 
setting, as well as a knowledge of the spectral 

15 response of the human eye. The resulting spectral 

curves are referred to as the color-matching functions 
(CMFs) for the display. 

It is desirable to have the camera exhibit 
spectral sensitivities only in the visible portion of 

20 electro-magnetic spectriim (approximately 400 to 700 
nm. ) . In addition, it is desirable that the overall 
spectral sensitivities of the camera correspond to a 
set of all-positive color-matching-f unctions (CMFs), 
If these requirements are met, the camera will be able 

25 to discern color information in the scene in much the 
same way that a human observer does . Failure to 
achieve this goal will result in color reproduction 
errors. (This failure mechanism is referred to as 
metamerism. ) 

30 A set of spectral curves is defined as a set 

of CMFs if, and only if, it can be exactly derived from 
the spectral response of the human eye via a linear 3x3 
transformation. An infinite number of CMFs are 
possible according to this definition. The CIE 

35 (Commission Internationale De L'Eclairage) has 

published standardized spectral data sets describing 
the response of the human eye. This data may be found 





in CIE publication 15.2 (1986) Colorimetry - Second 
Edition in table 2.5. Another useful feature of CMFs 
is the fact that any two sets of CMFs are directly 
related to each other through a unique 3x3 linear 
5 transformation. 

One practical limitation in the selection of 
a set of CMFs for the camera is the restriction that 
they be all positive, whereas the CMFs describing a 
color CRT typically have negative lobes. This is not a 

10 problem in practice since a linear 3x3 transformation 
may be employed, as discussed above, to correct the 
camera's output color signals for rendition on the CRT 
display. This linear 3x3 transformation is often 
referred to in the art as a color-correction matrix. 

15 Another practical restriction in the selection of a set 
of camera CMF's is the need to minimize the size of the 
off-diagonal coefficients in the color-correction 
matrix since these are directly responsible for 
degrading the noise performance of the imaging system. 

20 The optical path of an electronic camera may 

consist of various components--each with its own 
spectral characteristics. Among these components one 
would ordinarily find a lens, blur-filter, infra-red 
cut-off filter and a CCD imager. The overall spectral 

25 sensitivity of the camera is determined by the combined 
spectral responses of the individual components. 
Figure 1 illustrates the spectral characteristics for a 
typical color CCD camera including the combined effects 
of all of the optical components. These curves have 

30 been normalized to unit response for comparison 

purposes as is the standard practice when working with 
color-matching functions . 

Included in Figure 1 is a second set (dotted 
lines) of curves representing the transformed spectral 

35 characteristics of the camera following the color- 
correction matrix operation. Note that the transformed 
spectral responses have negative lobes whereas the 
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original camera spectral responses do not. Figure 2 
compares the transformed spectral responses of the 
camera (dotted lines) with the CMFs for a CRT having 
CCIR Rec . 709 phosphors and a 6500 Kelvin white point. 
5 It can be seen that elements in a real camera have 

errors in spectral response that prevent replication of 
CMFs regardless of the transformation. Errors are 
normally spread among all colors in a way that 
minimizes color errors, but the result inevitably is 
10 not a perfect match, as seen particularly in the 

transformed camera red spectral response in Figure 2 . 



in U.S. Patent 5,253,047, in which a color temperature 
detecting circuit modifies the matrix coefficients for 



15 a primary color separator used to perform a color- 
correction operation for a color video camera. The 
primary color separator is used to compute the red, 
green and blue primary color signals for the 
luminance /chrominance signals generated by the camera 

20 detector circuitry. In U.S. Patent Application Serial 
No. 08/569,645, "Method and Apparatus for Color- 
Correcting Multi-channel Signals of a Digital Camera", 
filed December 8, 1995 to Spaulding et al, an improved 
method is used to select the color-correction matrix 

25 coefficients to account for changes in illuminant color 
temperature. In particular, this method provides 
optimum compensation for variations in the scene 
illuminant by using all of the degrees-of -freedom 
available in the primary color separator matrix. 

30 A color-correction matrix is shown in U.S. 

Patent No. 5,001,663 as one component of a digital- 
signal processing chipset for a high performance 
digital color video camera. The implementation 
illustrated requires that the matrix be mask-programmed 

35 into the chip during fabrication. This approach fixes 
the matrix coefficients during the production process 
such that color correction is specific to a defined 



The use of a color-correction matrix is shown 
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type, or family, of cameras. This is ordinarily done 
by establishing the matrix coefficients to account for 
the optical component spectral characteristic or 
illuminant color temperature of a defined reference 
5 camera, and then embodying these coefficients in each 
manufactured camera. 



example of this approach, describing improved 
resolution and reproduction of hard copies made from 

10 images captured by different types of electronic still 
cameras. Subtractive- type color processing is used to 
attempt to stabilize the primaries associated with 
image dyes used to produce the hard copy images, 
preceded by additive-type processing which attempts to 

15 correct the camera sensitivities appropriately for the 
stabilized primaries. The additive- type color 
processing may be in the camera itself to ensure that 
each output device achieves optimism color reproduction 
from signals corresponding to those provided by a 

20 defined reference camera. This arrangement allows 
signals from different types of cameras, i.e., 
corresponding to different defined reference cameras 
(e.g., high resolution professional cameras vs. low 
resolution consumer cameras) , to provide input to 

25 different types of hardcopy devices and media. 



proliferate in the marketplace, there is increased need 
that images from comparable cameras or scanners produce 
comparable colors to the human observer. 

30 Unfortunately, small variations in optical component 
spectral characteristics, even within the same family 
of cameras, can produce noticeable color differences in 
the output images. Heretofore, the approaches taken do 
not account for variations in optical component 

35 spectral characteristics from individual imaging device 
to individual imaging device. 



U.S. Patent No. 5,189,511 is a further 



As digital cameras and low cost scanners 



SUMMARY OF THE INVENTION 

The aforementioned problems are solved with a 
5 technique for optimum color correction utilizing 

customized matrix coefficients for a particular imaging 
device. According to the invention, a digital imaging 
device captures an image and generates a color signal 
from the image for application to an output device 

10 having specific color sensitivities, the imaging device 
further being one of many devices of the same type 
useful with the output device. The digital imaging 
device, for example a digital camera, includes a color 
sensor for capturing the image and generating a color 

15 signal from the captured image, the color sensor having 
predetermined spectral sensitivities, and an optical 
section that is interposed in the image light directed 
to the color sensor, the optical section also having 
predetermined spectral characteristics . Importantly, 

20 the combination of the spectral sensitivities of the 
color sensor and the spectral characteristics of the 
optical section uniquely distinguish this particular 
imaging device from other imaging devices of the same 
type. By providing a set of matrix coefficients 

25 uniquely determined for this imaging device, the matrix 
coefficients optimally correct the spectral 
sensitivities of the color sensor and the spectral 
characteristics of the optical section for the color 
sensitivities of the output device. 

30 Besides providing an optimal level of color 

correction, the present invention has the advantage 
that the color reproduction variation from one camera 
to the next is accordingly minimized. This reduces the 
occurrence of color nonunif ormity between the same 

35 digital images captured by different digital cameras. 

These and other aspects, objects, features 
and advantages of the present invention will be more 




clearly understood and appreciated from a review of the 
following detailed description of the preferred 
embodiments and appended claims, and by reference to 
the accompanying drawings . 

5 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a comparison of the normalized 
original camera spectral characteristics and the 

10 transformed spectral characteristics following 
application of a color-correction matrix. 

FIG. 2 is a comparison of the transformed 
camera spectral characteristics from Figure 1 and the 
actual CMFs for a particular CRT. 

15 FIG. 3 is a block diagram showing the 

preferred embodiment of the invention wherein the color 
matrix is located on the optical assembly of the 
camera, but the actual color-correction operation 
utilizing the matrix is performed external to the 

20 camera. 

FIG. 4 is a block diagram showing a second 
version of the preferred embodiment of the invention 
wherein the color matrix is located on the optical 
assembly of the camera and the color-correction 

25 operation is performed internal to the camera. 

FIG, 5 is a block diagram showing a third 
embodiment of the invention wherein the color matrix is 
located within the camera but external to the optical 
assembly of the camera. 

30 FIG. 6 is a block diagram showing a fourth 

embodiment of the invention wherein the lens is 
removable from the camera. 

FIG. 7 is a block diagram outlining a general 
method for obtaining the color-correction matrix 

35 coefficients. 
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FIG. 8 is a block diagram of an electronic 
imaging system incorporating an imaging device in 
accordance with the invention. 



DETAILED DESCRIPTION OF THE INVENTION 



Because electronic imaging devices employing 



electronic sensors are well known, the present 
description will be directed in particular to elements 

10 forming part of, or cooperating more directly with, 
apparatus in accordance with the present invention. 
Elements not specifically shown or described herein may 
be selected from those known in the art. Certain 
aspects of the embodiments to be described may be 

15 provided in software. Given the system description as 
described in the following materials, all such software 
implementation is conventional and within the ordinary 
skill in such arts. 



20 imaging device is a device which uses an electronic 
sensor to capture an image either directly from an 
object or indirectly from a medium, such as film; 
signal processing to represent the captured signal 
numerically; and some storage device to preserve the 

25 numerical image data. It is further known for a 

digital imaging device, particularly a digital camera, 
to use a removable storage device, such as an 
integrated circuit memory card, to store images. For 
instance, U.S. Patent No. 5,016,107 describes an 

30 electronic still camera utilizing image compression and 
providing digital storage in a removable memory card 
having a static random access memory. In this camera, 
the integrated circuits in the removable memory card 
store image data and a directory locating the data. 

35 The image data provided by the digital imaging device 
and stored in a memory card is ordinarily used to 
produce some type of display or print, for example, a 



As understood in the prior art, a digital 



CRT display or a digital print made from images scanned 
from film or taken by an electronic camera. 

As shown in Figure 8, the digital imaging 
device, which may be an electronic camera 10 or a 
5 scanner 11, is utilized in a system including a digital 
processor 12 . The digital image information produced 
by the digital imaging device is downloaded to the 
peripheral digital processor 12, as shown in Figure 8, 
for further processing into a digital image. The 

10 downloading can be accomplished for either device in a 
number of ways, for instance by a cable connection 13 
through an interface 14, or by removable media, such as 
a memory card 15, through a card reader 16. A suitable 
color CRT display 17 is connected to the digital 

15 processor 12 for displaying the images, and a printer* 
18 is connected to print out copies 18a of the images. 
A keyboard 19 is also connected for use in the 
processing of the images. The digital processor 12, 
which can be part of a conventional programmed 

20 computer, utilizes conventional processing techniques 
to process the digital image information according to 
algorithms stored in the computer or provided by 
application software used with the computer. For 
example, the digital processor 12 may include a 

25 conventional color management system, which links the 
input device (camera 10 or scanner 11) and the output 
device {CRT 17 or printer 18) by utilizing device 
profiles appropriate for the type of input and output 
devices used (e.g., one input profile for the camera 10 

30 and another input profile for the scanner 11) . 

The ultimate color performance of an 
electronic camera is directly influenced by the various 
optical components which comprise the image capture 
path. It is possible to maximize the color 

35 reproduction accuracy of a particular camera by 

computing a unique color correction matrix for that 
camera which compensates for the unique optical 




characteristic of that camera. This approach also 
minimizes the variation in color reproduction from one 
camera to the next. With reference to Figure 3, a 
simplified block diagram illustrating the preferred 
5 embodiment of the invention is shown. The electronic 
camera 10 has an optical sub-assembly 2 0 containing a 
lens 22, an infrared cutoff filter 24, a blur filter 26 
and a CCD imager 2 8 with an integral color filter array 
(CFA) 30. The optical subassembly 20 has predetermined 

10 spectral characteristics, comprising the combination of 
the spectral sensitivities of the CCD imager 2 8 and the 
spectral characteristics of the lens 22, the infrared 
cutoff filter 24, and the blur filter 26. Due to these 
spectral sensitivities and spectral characteristics, 

15 the combination thereof uniquely distinguish this 

imaging device from other imaging devices of the same 
type. In other words, although different cameras 10 
contain nominally identical optical elements, including 
sensors, their overall spectral responses will differ 

20 from camera to camera. 

The optical subassembly 20, which is used to 
capture an image of a scene 32, is designed to be 
removable from the camera 10 for purposes of servicing 
and calibration. When installed in the camera, the 

25 optical subassembly electrically connects to a 
preprocessing section 34 through an electrical 
connecting means 35. Image-wise signals Si - Sn from 
the CCD imager 2 8 are converted to digital, linear RGB 
format within the camera by the pre-processing section 

30 34 using techniques and components familiar to those 
skilled in the art. These digital RGB signals 
represent the red, green and blue primary components of 
the image, respectively. Pre-processing section 3 4 may 
perform such well-known tasks as double-correlated 

35 sampling of the CCD signals, black-level control, 

white-balance , analog-to-digital conversion, conversion 




of the CCD signals to RGB and interpolation of the CFA 
data to produce RGB values at each pixel location. 

Digital RGB values from the pre-processing 
section 34 are transformed to a set of color-corrected 
5 RGB values (Rcc. Gcc/ Bcc) suitable for display on the 
color CRT display 17 by processing in a color matrix 
operation 40. In this embodiment of the invention, the 
color-correction matrix 40 operation is performed 
external to the camera as is shown in Figure 3 in, for 

10 example, the digital processor 12 shown in Figure 8. 

Therefore, the RGB signals and the matrix coefficients 
are provided to the external digital processor 12 via 
interface lines 37a and b. In a .second version of the 
preferred embodiment of this invention, the color- 

15 correction matrix 40 operation is performed internal to 
the camera as illustrated in Figure 4. Where this step 
is performed is not important to the teaching of this 
invention. In either case the color-correction matrix 
40 operation is performed on RGB signals which vary 

20 linearly with exposure. 

Color-corrected RGB signals (Rcc/ Gcc/ Bcc) 
following the color-correction matrix 40 operation are 
converted to a format suitable for CRT display by a 
post-processing section 50 using techniques and 

25 components familiar to those skilled in the art. Such 
post processing operations may include such tasks as 
interpolation, edge-enhancement and tone- scale 
remapping, for example. 

In the preferred embodiment of this invention 

30 as shown in Figure 3 and 4, the color-correction matrix 
coefficients for the color matrix operation 40 are 
stored in a digital memory 36 co-located on the optical 
sub-assembly 2 0 with all of the other optical 
components. These coefficients are uniquely determined 

35 for each camera in order to correct the spectral 

sensitivities of the particular CCD imager 28 in the 
camera 10, and the spectral characteristics of the 




particular other elements in the optical subassembly 
20, for the color sensitivities of the type of output 
device being used. (For this reason, while 
representing a specific imaging device, the 
5 coefficients are ordinarily calculated in relation to a 
reference output device, rather than a specific 
individual output device.) These coefficients are then 
applied to the color matrix operation 40 for color 
correction of the capture image. This approach has 

10 advantages in the production and service environments. 
In the production environment, optical sub-assemblies 
2 0 can be fabricated, calibrated and stocked for later 
integration into the final product 10 without the need 
for calibrating the final product. In the service 

15 environment, since the optical subassembly is 

replaceably interconnected to the preprocessing section 
34 through the electrical connecting means 35, optical 
sub-assemblies 20 can be simply replaced without the 
need for calibrating the repaired product. Since each 

20 optical subassembly 2 0 is calibrated for the particular 
optical components on the subassembly,' it may be 
appreciated that the matrix coefficients stored in the 
memory 3 6 are unique for each subassembly 20, and 
therefore for each camera 10 . 

25 Note that, although RGB signals have been 

discussed by way of example as the tristimulus format 
of choice for representing the scene color information, 
this invention is not restricted to use with this 
format alone. Other tristimulus formats such as the 

30 CIE XYZ format are equally applicable and may, in fact, 
present advantages in a particular implementation. 
Since the CIE XYZ format is a device independent space 
based on a set of CMFs defined by the CIE 1931 Standard 
Colorimetric Observer (2"^) , the matrix coefficients 

35 could be used to generate an input profile unique to 
each camera which will correct the spectral 
sensitivities of the camera for the standardized CMFs 





of this device independent space. For example, in one 
application following the ICC Profile Format 
Specification (Version 3.2, November 20, 1995, 
published by the International Color Consortium) , RGB 
5 input profiles are es.tablished which will correct the 
spectral sensitivities of an input device for a 
connection space. Thereupon, output profiles are used 
to convert the signals from the connection space to a 
format that is expected by an output device. It should 

10 therefore be understood that this invention encompasses 
linear tristimulus formats in general while the 
discussion is limited to the familiar RGB format for 
ease of understanding. 

Figure 5 illustrates a third embodiment of 

15 the invention wherein the color-correction matrix 

coefficients for the matrix operation 40 are stored in 
a digital memory 3 8 located somewhere within the camera 
10 but not necessarily on the optical sub-assembly 20. 
In this embodiment, the aforementioned advantages in 

20 the production and service environments are not 

realizable but the color reproduction accuracy and 
consistency goals of the invention are not compromised. , 

Figure 6 illustrates a fourth embodiment of 
the invention wherein the lens 22 is not physically 

25 part of the optical sub-assembly and may, in fact, be 
removable as is the case of a camera with 
interchangeable lenses. In this situation the camera 
or optical sub-assembly, depending on where the color- 
coefficient matrix digital memory 3 8 is located, is 

30 calibrated using a lens with a spectral response 

representing the average of the spectral responses of 
all lenses that are anticipated to be used with the 
camera. This represents only a slight compromise since 
the lens does not contribute significantly to 

35 variations in color reproduction performance. 

The present invention has been described thus 
far with regard to a series of preferred embodiments 
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outlining the location of the color-correction matrix 
coefficients within the camera as well as their 
application in the image processing path. Figure 7 
illustrates a general method for obtaining these 
5 coefficients. A color chart 72 containing colors 
representative of the expected range of real world 
colors provides the basis for the coefficients 
calculation process. An example of such a chart is the 
well known Macbeth Color Checker. 

10 A necessary first step involves measurement 

of the spectral characteristic (S;^) of the illuminant 
74 as well as measurement of the spectral 
characteristic (Ri;^) of each color patch of the color 
chart 72. CIE tristimulus values (XYZ) , describing the 

15 location of a particular color patch in the 

3 -dimensional XYZ color space, may then be computed 
from the measured spectral data as described in the 
aforementioned CIE publication 15.2 (1986) 
ColoriiriGtry-- Second Edition on pages 22-23. This set 

20 of XYZ values becomes the colorimetric aims for the 
camera. 



25 the captured color chart image are next subjected to a 
normalization process 78 to provide RGB signals which 
vary linearly with scene luminance. The normalization 
process 78 may include removal of the camera gamma/knee 
characteristic as well as black-level, white-balance 

30 and exposure errors . 



of color-corrected RGB signals by color-correction 
matrix transformation 79 . Initially this matrix is set 
equal to the identity matrix. The coefficients are 
35 subsequently adjusted in an iterative fashion by a 

regression process 80 until the average color error for 



The electronic camera 10 is used to capture 
an image of the color chart 72 under the specified 
illuminant 74. RGB camera signals representative of 



Linear RGB signals are transformed to a set 




all color patches of the color chart 72 is reduced to a 
predetermined level . 

Color-corrected RGB signals from color- 
correction matrix transformation 79 are retransf ormed 
5 to a set of CIE XYZ signals by CRT phosphor matrix 

transformation 84. These signals represent the colors 
which would appear on the face of a reference CRT when 
presented with the color-corrected RGB signals from 
color-correction matrix transformation 79. It would be 

10 understood by someone skilled in the art that these 

color-corrected signals would first need to be modified 
to account for the nonlinear characteristic of the CRT 
phosphors. The resulting XYZ signals from CRT phosphor 
transformation matrix 84 represent the reproduced 

15 colors for each color patch of the color chart 72 for a 
reference output device, in this case the reference 
CRT. 

An error calculation process 82 determines 
the average error between the aim and reproduction 

20 signals for all of the color patches on the color chart 
72. An individual color error is first computed for 
each color patch using the square-root of the sum of 
the squares of the differences of the aim and 
reproduction X, Y and Z signals. This represents the 

25 vector length between the location of the aim and 
reproduction colors in the 3 -dimensional XYZ color 
space. The last part of the error calculation is to 
average the individual color errors . This average 
color error is used in the regression process 80. 

30 The foregoing matrix coefficient calculation 

process has been described from the standpoint of 
obtaining a good colorimetric match between the 
original scene and its reproduction on a CRT. For some 
applications this may not be the desired color 

35 reproduction goal. Modifications to the method shown 
in Figure 7 may be made to achieve a preferred color 
reproduction goal by taking into account such factors 



as chromatic adaptation and more perceptually uniform 
color spaces . 

The invention has been described with 
reference to a preferred embodiment. However, it will 
be appreciated that variations and modifications can be 
effected by a person of ordinary skill in the art 
without departing from the scope of the invention. 
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PARTS LIST 

10 Electronic camera 

11 Electronic scanner 

12 Image processing external to camera (peripheral 

digital processor) 

13 Interface cable 

14 Digital interface 

15 Memory card, removable 

16 Card reader 

17 Color CRT display 

18 Printer, color 
18a Color print 

19 Keyboard 

2 0 Optical sub-assembly 
22 Lens 

24 Infra-red cutoff filter 

26 Blur filter 

28 CCD Imager 

30 Color filter array (CPA) 

3 2 Scene 

34 Camera video pre-processing section 

36 Color-correction matrix coefficient memory on 

op t i c a 1 s uba s s embly 

3 8 Color-correction matrix coefficient memoiry in 

camera 

40 Color-correction matrix transformation 

50 Post-processing section 

72 Color chart 

74 Illuminant 

76 XYZ calculation process 

78 Normalization process 

79 Color-correction matrix transformation 

80 Regression process 

82 Color error calculation process 



